

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Testing - Integration Tests - Introduction &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
        <script src="../../../../_static/jquery.js"></script>
        <script src="../../../../_static/underscore.js"></script>
        <script src="../../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../../genindex/" />
    <link rel="search" title="Search" href="../../../../search/" />
    <link rel="next" title="Integration Tests using Teuthology Workflow" href="../tests-integration-testing-teuthology-workflow/" />
    <link rel="prev" title="Teuthology User Guide" href="../" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../">向 Ceph 贡献：开发者指南</a> &raquo;</li>
        
          <li><a href="../">Teuthology User Guide</a> &raquo;</li>
        
      <li>Testing - Integration Tests - Introduction</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../../_sources/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../../">
          

          
            
            <img src="../../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../architecture/">体系结构</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">开发者指南</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../intro/">简介</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../essentials/">必备知识</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../merging/">何时、合并了什么</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../issue-tracker/">问题追踪器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../basic-workflow/">基本工作流</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../tests-unit-tests/">测试：单元测试</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">测试：集成测试(Teuthology)</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">Introduction</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#teuthology-consumes-packages">Teuthology consumes packages</a></li>
<li class="toctree-l4"><a class="reference internal" href="#the-nightlies">The Nightlies</a></li>
<li class="toctree-l4"><a class="reference internal" href="#testing-priority">Testing Priority</a></li>
<li class="toctree-l4"><a class="reference internal" href="#suites-inventory">Suites Inventory</a></li>
<li class="toctree-l4"><a class="reference internal" href="#teuthology-describe">teuthology-describe</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-to-run-integration-tests">How to run integration tests</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-integration-tests-are-defined">How integration tests are defined</a></li>
<li class="toctree-l4"><a class="reference internal" href="#reading-a-standalone-test">Reading a standalone test</a></li>
<li class="toctree-l4"><a class="reference internal" href="#test-descriptions">Test descriptions</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-tests-are-built-from-directories">How tests are built from directories</a></li>
<li class="toctree-l4"><a class="reference internal" href="#filtering-tests-by-their-description">Filtering tests by their description</a></li>
<li class="toctree-l4"><a class="reference internal" href="#reducing-the-number-of-tests">Reducing the number of tests</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-workflow/">Workflow</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-debugging-tips/">Debugging Tips</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-sentry-developers-guide/">Sentry Notes</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../running-tests-locally/">测试：在本地运行测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dash-devel/">Ceph Dashboard 开发者文档 (之前是 HACKING.rst)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../jaegertracing/">Tracing 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../cephadm/">Cephadm 开发者文档</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="testing-integration-tests-introduction">
<span id="tests-integration-testing-teuthology-intro"></span><h1>Testing - Integration Tests - Introduction<a class="headerlink" href="#testing-integration-tests-introduction" title="Permalink to this headline">¶</a></h1>
<p>Ceph has two types of tests: <a class="reference internal" href="../../tests-unit-tests/#make-check"><span class="std std-ref">make check</span></a> tests and
integration tests. When a test requires multiple machines, root access, or lasts
for a long time (for example, to simulate a realistic Ceph workload), it is
deemed to be an integration test. Integration tests are organized into “suites”,
which are defined in the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a> and run with the
<code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command is part of the <a class="reference external" href="https://github.com/ceph/teuthology">teuthology framework</a>.
In the sections that follow we attempt to provide a detailed introduction
to that framework from the perspective of a beginning Ceph developer.</p>
<div class="section" id="teuthology-consumes-packages">
<h2>Teuthology consumes packages<a class="headerlink" href="#teuthology-consumes-packages" title="Permalink to this headline">¶</a></h2>
<p>It may take some time to understand the significance of this fact, but it
is <cite>very</cite> significant. It means that automated tests can be conducted on
multiple platforms using the same packages (RPM, DEB) that can be
installed on any machine running those platforms.</p>
<p>Teuthology has a <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/distros/supported">list of platforms that it supports</a> (as of
September 2020 the list consisted of “RHEL/CentOS 8” and “Ubuntu 18.04”). It
expects to be provided pre-built Ceph packages for these platforms.  Teuthology
deploys these platforms on machines (bare-metal or cloud-provisioned), installs
the packages on them, and deploys Ceph clusters on them - all as called for by
the test.</p>
</div>
<div class="section" id="the-nightlies">
<h2>The Nightlies<a class="headerlink" href="#the-nightlies" title="Permalink to this headline">¶</a></h2>
<p>A number of integration tests are run on a regular basis in the <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php">Sepia
lab</a> against the official Ceph repositories (on the <code class="docutils literal notranslate"><span class="pre">master</span></code> development
branch and the stable branches). Traditionally, these tests are called “the
nightlies” because the Ceph core developers used to live and work in
the same time zone and from their perspective the tests were run overnight.</p>
<p>The results of nightly test runs are published at <a class="reference external" href="http://pulpito.ceph.com/">http://pulpito.ceph.com/</a>
under the user <code class="docutils literal notranslate"><span class="pre">teuthology</span></code>. The developer nick appears in URL of the the
test results and in the first column of the Pulpito dashboard.  The results are
also reported on the <a class="reference external" href="https://ceph.com/irc/">ceph-qa mailing list</a>.</p>
</div>
<div class="section" id="testing-priority">
<h2>Testing Priority<a class="headerlink" href="#testing-priority" title="Permalink to this headline">¶</a></h2>
<p>In brief: in the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command option <code class="docutils literal notranslate"><span class="pre">-p</span> <span class="pre">&lt;N&gt;</span></code>, set the value of <code class="docutils literal notranslate"><span class="pre">&lt;N&gt;</span></code> to a number lower than 1000. An explanation of why follows.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command includes an option <code class="docutils literal notranslate"><span class="pre">-p</span> <span class="pre">&lt;N&gt;</span></code>. This option specifies the priority of the jobs submitted to the queue. The lower the value of <code class="docutils literal notranslate"><span class="pre">N</span></code>, the higher the priority.</p>
<p>The default value of <code class="docutils literal notranslate"><span class="pre">N</span></code> is <code class="docutils literal notranslate"><span class="pre">1000</span></code>. This is the same priority value given to the nightly tests (the nightlies). Often, the volume of testing done during the nightly tests is so great that the full number of nightly tests do not get run during the time allotted for their run.</p>
<p>Set the value of <code class="docutils literal notranslate"><span class="pre">N</span></code> lower than <code class="docutils literal notranslate"><span class="pre">1000</span></code>, or your tests will not have priority over the nightly tests. This means that they might never run.</p>
<p>Select your job’s priority (the value of <code class="docutils literal notranslate"><span class="pre">N</span></code>) in accordance with the following guidelines:</p>
<table class="colwidths-given docutils align-default">
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Priority</p></th>
<th class="head"><p>Explanation</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><strong>N &lt; 10</strong></p></td>
<td><p>Use this if the sky is falling and some group of tests must be run ASAP.</p></td>
</tr>
<tr class="row-odd"><td><p><strong>10 &lt;= N &lt; 50</strong></p></td>
<td><p>Use this if your tests are urgent and blocking other important
development.</p></td>
</tr>
<tr class="row-even"><td><p><strong>50 &lt;= N &lt; 75</strong></p></td>
<td><p>Use this if you are testing a particular feature/fix and running fewer
than about 25 jobs. This range is also used for urgent release testing.</p></td>
</tr>
<tr class="row-odd"><td><p><strong>75 &lt;= N &lt; 100</strong></p></td>
<td><p>Tech Leads regularly schedule integration tests with this priority to
verify pull requests against master.</p></td>
</tr>
<tr class="row-even"><td><p><strong>100 &lt;= N &lt; 150</strong></p></td>
<td><p>This priority is used for QE validation of point releases.</p></td>
</tr>
<tr class="row-odd"><td><p><strong>150 &lt;= N &lt; 200</strong></p></td>
<td><p>Use this priority for 100 jobs or fewer that test a particular feature
or fix.  Results are available in about 24 hours.</p></td>
</tr>
<tr class="row-even"><td><p><strong>200 &lt;= N &lt; 1000</strong></p></td>
<td><p>Use this priority for large test runs.  Results are available in about a
week.</p></td>
</tr>
</tbody>
</table>
<p>To see how many jobs the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command will trigger, use the
<code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag. If you are happy with the number of jobs returned by the
dry run, issue the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command again without <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> and
with <code class="docutils literal notranslate"><span class="pre">-p</span></code> and an appropriate number as an argument.</p>
<p>To skip the priority check, use <code class="docutils literal notranslate"><span class="pre">--force-priority</span></code>. Be considerate of the needs of other developers to run tests, and use <code class="docutils literal notranslate"><span class="pre">--force-priority</span></code> only in emergencies.</p>
</div>
<div class="section" id="suites-inventory">
<h2>Suites Inventory<a class="headerlink" href="#suites-inventory" title="Permalink to this headline">¶</a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">suites</span></code> directory of the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a> contains all the
integration tests for all the Ceph components.</p>
<table class="docutils align-default" id="id2">
<caption><span class="caption-text"><strong>Suites</strong></span><a class="headerlink" href="#id2" title="Permalink to this table">¶</a></caption>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p><strong>Component</strong></p></td>
<td><p><strong>Function</strong></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy">ceph-deploy</a></p></td>
<td><p>install a Ceph cluster with <code class="docutils literal notranslate"><span class="pre">ceph-deploy</span></code> (<a class="reference external" href="../../../../man/8/ceph-deploy">ceph-deploy man page</a>)</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/dummy">dummy</a></p></td>
<td><p>get a machine, do nothing and return success (commonly used to verify
that the integration testing infrastructure works as expected)</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/fs">fs</a></p></td>
<td><p>test CephFS mounted using kernel and FUSE clients, also with multiple MDSs.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/krbd">krbd</a></p></td>
<td><p>test the RBD kernel module</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/powercycle">powercycle</a></p></td>
<td><p>verify the Ceph cluster behaves when machines are powered off and on
again</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rados">rados</a></p></td>
<td><p>run Ceph clusters including OSDs and MONs, under various conditions of
stress</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rbd">rbd</a></p></td>
<td><p>run RBD tests using actual Ceph clusters, with and without qemu</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rgw">rgw</a></p></td>
<td><p>run RGW tests using actual Ceph clusters</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/smoke">smoke</a></p></td>
<td><p>run tests that exercise the Ceph API with an actual Ceph cluster</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/teuthology">teuthology</a></p></td>
<td><p>verify that teuthology can run integration tests, with and without OpenStack</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/upgrade">upgrade</a></p></td>
<td><p>for various versions of Ceph, verify that upgrades can happen without disrupting an ongoing workload</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="teuthology-describe">
<h2>teuthology-describe<a class="headerlink" href="#teuthology-describe" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">teuthology-describe</span></code> was added to the <a class="reference external" href="https://github.com/ceph/teuthology">teuthology framework</a> to facilitate
documentation and better understanding of integration tests.</p>
<p>Tests can be documented by embedding <code class="docutils literal notranslate"><span class="pre">meta:</span></code> annotations in the yaml files
used to define the tests. The results can be seen in the <a class="reference external" href="https://gist.github.com/jdurgin/09711d5923b583f60afc">teuthology-desribe
usecases</a></p>
<p>Since this is a new feature, many yaml files have yet to be annotated.
Developers are encouraged to improve the coverage and the quality of the
documentation.</p>
</div>
<div class="section" id="how-to-run-integration-tests">
<h2>How to run integration tests<a class="headerlink" href="#how-to-run-integration-tests" title="Permalink to this headline">¶</a></h2>
<p>Typically, the <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php">Sepia lab</a> is used to run integration tests. But as a new Ceph
developer, you will probably not have access to the <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php">Sepia lab</a>.  You might
however be able to run some integration tests in an environment separate from
the <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php">Sepia lab</a> . Ask members from the relevant team how to do this.</p>
<p>One way to run your own integration tests is to set up a teuthology cluster on
bare metal. Setting up a teuthology cluster on bare metal is a complex task.
Here are <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/LAB_SETUP.html">some notes</a> to get
you started if you decide that you are interested in undertaking the complex
task of setting up a teuthology cluster on bare metal.</p>
<p>Running integration tests on your code contributions and publishing the results
allows reviewers to verify that changes to the code base do not cause
regressions, and allows reviewers to analyze test failures when they occur.</p>
<p>Every teuthology cluster, whether bare-metal or cloud-provisioned, has a
so-called “teuthology machine” from which tests suites are triggered using the
<code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command.</p>
<p>A detailed and up-to-date description of each <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/commands/teuthology-suite.html">teuthology-suite</a> option is
available by running the following command on the teuthology machine:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">teuthology-suite --help</span>
</pre></div></div></div>
<div class="section" id="how-integration-tests-are-defined">
<h2>How integration tests are defined<a class="headerlink" href="#how-integration-tests-are-defined" title="Permalink to this headline">¶</a></h2>
<p>Integration tests are defined by yaml files found in the <code class="docutils literal notranslate"><span class="pre">suites</span></code>
subdirectory of the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a> and implemented by python
code found in the <code class="docutils literal notranslate"><span class="pre">tasks</span></code> subdirectory. Some tests (“standalone tests”)
are defined in a single yaml file, while other tests are defined by a
directory tree containing yaml files that are combined, at runtime, into a
larger yaml file.</p>
</div>
<div class="section" id="reading-a-standalone-test">
<span id="reading-standalone-test"></span><h2>Reading a standalone test<a class="headerlink" href="#reading-a-standalone-test" title="Permalink to this headline">¶</a></h2>
<p>Let us first examine a standalone test, or “singleton”.</p>
<p>Here is a commented example using the integration test
<a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/admin-socket.yaml">rados/singleton/all/admin-socket.yaml</a></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">roles</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">mon.a</span>
  <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">osd.0</span>
  <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">osd.1</span>
<span class="nt">tasks</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">install</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">ceph</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">admin_socket</span><span class="p">:</span>
    <span class="nt">osd.0</span><span class="p">:</span>
      <span class="nt">version</span><span class="p">:</span>
      <span class="nt">git_version</span><span class="p">:</span>
      <span class="nt">help</span><span class="p">:</span>
      <span class="nt">config show</span><span class="p">:</span>
      <span class="nt">config set filestore_dump_file /tmp/foo</span><span class="p">:</span>
      <span class="nt">perf dump</span><span class="p">:</span>
      <span class="nt">perf schema</span><span class="p">:</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">roles</span></code> array determines the composition of the cluster (how
many MONs, OSDs, etc.) on which this test is designed to run, as well
as how these roles will be distributed over the machines in the
testing cluster. In this case, there is only one element in the
top-level array: therefore, only one machine is allocated to the
test. The nested array declares that this machine shall run a MON with
id <code class="docutils literal notranslate"><span class="pre">a</span></code> (that is the <code class="docutils literal notranslate"><span class="pre">mon.a</span></code> in the list of roles) and two OSDs
(<code class="docutils literal notranslate"><span class="pre">osd.0</span></code> and <code class="docutils literal notranslate"><span class="pre">osd.1</span></code>).</p>
<p>The body of the test is in the <code class="docutils literal notranslate"><span class="pre">tasks</span></code> array: each element is
evaluated in order, causing the corresponding python file found in the
<code class="docutils literal notranslate"><span class="pre">tasks</span></code> subdirectory of the <a class="reference external" href="https://github.com/ceph/teuthology">teuthology repository</a> or
<a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a> to be run. “Running” in this case means calling
the <code class="docutils literal notranslate"><span class="pre">task()</span></code> function defined in that file.</p>
<p>In this case, the <a class="reference external" href="https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py">install</a>
task comes first. It installs the Ceph packages on each machine (as
defined by the <code class="docutils literal notranslate"><span class="pre">roles</span></code> array). A full description of the <code class="docutils literal notranslate"><span class="pre">install</span></code>
task is <a class="reference external" href="https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py">found in the python file</a>
(search for “def task”).</p>
<p>The <code class="docutils literal notranslate"><span class="pre">ceph</span></code> task, which is documented <a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/tasks/ceph.py">here</a> (again,
search for “def task”), starts OSDs and MONs (and possibly MDSs as well)
as required by the <code class="docutils literal notranslate"><span class="pre">roles</span></code> array. In this example, it will start one MON
(<code class="docutils literal notranslate"><span class="pre">mon.a</span></code>) and two OSDs (<code class="docutils literal notranslate"><span class="pre">osd.0</span></code> and <code class="docutils literal notranslate"><span class="pre">osd.1</span></code>), all on the same
machine. Control moves to the next task when the Ceph cluster reaches
<code class="docutils literal notranslate"><span class="pre">HEALTH_OK</span></code> state.</p>
<p>The next task is <code class="docutils literal notranslate"><span class="pre">admin_socket</span></code> (<a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/tasks/admin_socket.py">source code</a>).
The parameter of the <code class="docutils literal notranslate"><span class="pre">admin_socket</span></code> task (and any other task) is a
structure which is interpreted as documented in the task. In this example
the parameter is a set of commands to be sent to the admin socket of
<code class="docutils literal notranslate"><span class="pre">osd.0</span></code>. The task verifies that each of them returns on success (i.e.
exit code zero).</p>
<p>This test can be run with</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite rados/singleton/all/admin-socket.yaml fs/ext4.yaml</span>
</pre></div></div></div>
<div class="section" id="test-descriptions">
<h2>Test descriptions<a class="headerlink" href="#test-descriptions" title="Permalink to this headline">¶</a></h2>
<p>Each test has a “test description”, which is similar to a directory path,
but not the same. In the case of a standalone test, like the one in
<a class="reference internal" href="#reading-a-standalone-test">Reading a standalone test</a>, the test description is identical to the
relative path (starting from the <code class="docutils literal notranslate"><span class="pre">suites/</span></code> directory of the
<a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a>) of the yaml file defining the test.</p>
<p>Much more commonly, tests are defined not by a single yaml file, but by a
<cite>directory tree of yaml files</cite>. At runtime, the tree is walked and all yaml
files (facets) are combined into larger yaml “programs” that define the
tests. A full listing of the yaml defining the test is included at the
beginning of every test log.</p>
<p>In these cases, the description of each test consists of the
subdirectory under <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites">suites/</a> containing the
yaml facets, followed by an expression in curly braces (<code class="docutils literal notranslate"><span class="pre">{}</span></code>) consisting of
a list of yaml facets in order of concatenation. For instance the
test description:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span><span class="o">-</span><span class="n">deploy</span><span class="o">/</span><span class="n">basic</span><span class="o">/</span><span class="p">{</span><span class="n">distros</span><span class="o">/</span><span class="n">centos_7</span><span class="mf">.0</span><span class="o">.</span><span class="n">yaml</span> <span class="n">tasks</span><span class="o">/</span><span class="n">ceph</span><span class="o">-</span><span class="n">deploy</span><span class="o">.</span><span class="n">yaml</span><span class="p">}</span>
</pre></div>
</div>
<p>signifies the concatenation of two files:</p>
<ul class="simple">
<li><p>ceph-deploy/basic/distros/centos_7.0.yaml</p></li>
<li><p>ceph-deploy/basic/tasks/ceph-deploy.yaml</p></li>
</ul>
</div>
<div class="section" id="how-tests-are-built-from-directories">
<h2>How tests are built from directories<a class="headerlink" href="#how-tests-are-built-from-directories" title="Permalink to this headline">¶</a></h2>
<p>As noted in the previous section, most tests are not defined in a single
yaml file, but rather as a <cite>combination</cite> of files collected from a
directory tree within the <code class="docutils literal notranslate"><span class="pre">suites/</span></code> subdirectory of the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a>.</p>
<p>The set of all tests defined by a given subdirectory of <code class="docutils literal notranslate"><span class="pre">suites/</span></code> is
called an “integration test suite”, or a “teuthology suite”.</p>
<p>Combination of yaml facets is controlled by special files (<code class="docutils literal notranslate"><span class="pre">%</span></code> and
<code class="docutils literal notranslate"><span class="pre">+</span></code>) that are placed within the directory tree and can be thought of as
operators.  The <code class="docutils literal notranslate"><span class="pre">%</span></code> file is the “convolution” operator and <code class="docutils literal notranslate"><span class="pre">+</span></code>
signifies concatenation.</p>
<div class="section" id="convolution-operator">
<h3>Convolution operator<a class="headerlink" href="#convolution-operator" title="Permalink to this headline">¶</a></h3>
<p>The convolution operator, implemented as an empty file called <code class="docutils literal notranslate"><span class="pre">%</span></code>, tells
teuthology to construct a test matrix from yaml facets found in
subdirectories below the directory containing the operator.</p>
<p>For example, the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy/">ceph-deploy suite</a> is
defined by the <code class="docutils literal notranslate"><span class="pre">suites/ceph-deploy/</span></code> tree, which consists of the files and
subdirectories in the following structure</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>qa/suites/ceph-deploy
├── %
├── distros
│   ├── centos_latest.yaml
│   └── ubuntu_latest.yaml
└── tasks
    ├── ceph-admin-commands.yaml
    └── rbd_import_export.yaml
</pre></div>
</div>
<p>This is interpreted as a 2x1 matrix consisting of two tests:</p>
<ol class="arabic simple">
<li><p>ceph-deploy/basic/{distros/centos_7.0.yaml tasks/ceph-deploy.yaml}</p></li>
<li><p>ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml}</p></li>
</ol>
<p>i.e. the concatenation of centos_7.0.yaml and ceph-deploy.yaml and
the concatenation of ubuntu_16.04.yaml and ceph-deploy.yaml, respectively.
In human terms, this means that the task found in <code class="docutils literal notranslate"><span class="pre">ceph-deploy.yaml</span></code> is
intended to run on both CentOS 7.0 and Ubuntu 16.04.</p>
<p>Without the file percent, the <code class="docutils literal notranslate"><span class="pre">ceph-deploy</span></code> tree would be interpreted as
three standalone tests:</p>
<ul class="simple">
<li><p>ceph-deploy/basic/distros/centos_7.0.yaml</p></li>
<li><p>ceph-deploy/basic/distros/ubuntu_16.04.yaml</p></li>
<li><p>ceph-deploy/basic/tasks/ceph-deploy.yaml</p></li>
</ul>
<p>(which would of course be wrong in this case).</p>
<p>Referring to the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a>, you will notice that the
<code class="docutils literal notranslate"><span class="pre">centos_7.0.yaml</span></code> and <code class="docutils literal notranslate"><span class="pre">ubuntu_16.04.yaml</span></code> files in the
<code class="docutils literal notranslate"><span class="pre">suites/ceph-deploy/basic/distros/</span></code> directory are implemented as symlinks.
By using symlinks instead of copying, a single file can appear in multiple
suites. This eases the maintenance of the test framework as a whole.</p>
<p>All the tests generated from the <code class="docutils literal notranslate"><span class="pre">suites/ceph-deploy/</span></code> directory tree
(also known as the “ceph-deploy suite”) can be run with</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite ceph-deploy</span>
</pre></div></div><p>An individual test from the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy/">ceph-deploy suite</a> can be run by adding the
<code class="docutils literal notranslate"><span class="pre">--filter</span></code> option</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite <span class="se">\</span>
   --machine-type smithi <span class="se">\</span>
   --suite ceph-deploy/basic <span class="se">\</span>
   --filter <span class="s1">&#39;ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml}&#39;</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>To run a standalone test like the one in <a class="reference internal" href="#reading-a-standalone-test">Reading a standalone
test</a>, <code class="docutils literal notranslate"><span class="pre">--suite</span></code> alone is sufficient. If you want to run a single
test from a suite that is defined as a directory tree, <code class="docutils literal notranslate"><span class="pre">--suite</span></code> must
be combined with <code class="docutils literal notranslate"><span class="pre">--filter</span></code>. This is because the <code class="docutils literal notranslate"><span class="pre">--suite</span></code> option
understands POSIX relative paths only.</p>
</div>
</div>
<div class="section" id="concatenation-operator">
<h3>Concatenation operator<a class="headerlink" href="#concatenation-operator" title="Permalink to this headline">¶</a></h3>
<p>For even greater flexibility in sharing yaml files between suites, the
special file plus (<code class="docutils literal notranslate"><span class="pre">+</span></code>) can be used to concatenate files within a
directory. For instance, consider the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rbd/thrash">suites/rbd/thrash</a>
tree</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>qa/suites/rbd/thrash
├── %
├── clusters
│   ├── +
│   ├── fixed-2.yaml
│   └── openstack.yaml
└── workloads
    ├── rbd_api_tests_copy_on_read.yaml
    ├── rbd_api_tests.yaml
    └── rbd_fsx_rate_limit.yaml
</pre></div>
</div>
<p>This creates two tests:</p>
<ul class="simple">
<li><p>rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}</p></li>
<li><p>rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests.yaml}</p></li>
</ul>
<p>Because the <code class="docutils literal notranslate"><span class="pre">clusters/</span></code> subdirectory contains the special file plus
(<code class="docutils literal notranslate"><span class="pre">+</span></code>), all the other files in that subdirectory (<code class="docutils literal notranslate"><span class="pre">fixed-2.yaml</span></code> and
<code class="docutils literal notranslate"><span class="pre">openstack.yaml</span></code> in this case) are concatenated together
and treated as a single file. Without the special file plus, they would
have been convolved with the files from the workloads directory to create
a 2x2 matrix:</p>
<ul class="simple">
<li><p>rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}</p></li>
<li><p>rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests.yaml}</p></li>
<li><p>rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests_copy_on_read.yaml}</p></li>
<li><p>rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests.yaml}</p></li>
</ul>
<p>The <code class="docutils literal notranslate"><span class="pre">clusters/fixed-2.yaml</span></code> file is shared among many suites to
define the following <code class="docutils literal notranslate"><span class="pre">roles</span></code></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">roles</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="p p-Indicator">[</span><span class="nv">mon.a</span><span class="p p-Indicator">,</span> <span class="nv">mon.c</span><span class="p p-Indicator">,</span> <span class="nv">osd.0</span><span class="p p-Indicator">,</span> <span class="nv">osd.1</span><span class="p p-Indicator">,</span> <span class="nv">osd.2</span><span class="p p-Indicator">,</span> <span class="nv">client.0</span><span class="p p-Indicator">]</span>
<span class="p p-Indicator">-</span> <span class="p p-Indicator">[</span><span class="nv">mon.b</span><span class="p p-Indicator">,</span> <span class="nv">osd.3</span><span class="p p-Indicator">,</span> <span class="nv">osd.4</span><span class="p p-Indicator">,</span> <span class="nv">osd.5</span><span class="p p-Indicator">,</span> <span class="nv">client.1</span><span class="p p-Indicator">]</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">rbd/thrash</span></code> suite as defined above, consisting of two tests,
can be run with</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite rbd/thrash</span>
</pre></div></div><p>A single test from the rbd/thrash suite can be run by adding the
<code class="docutils literal notranslate"><span class="pre">--filter</span></code> option</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite <span class="se">\</span>
   --machine-type smithi <span class="se">\</span>
   --suite rbd/thrash <span class="se">\</span>
   --filter <span class="s1">&#39;rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}&#39;</span></span>
</pre></div></div></div>
</div>
<div class="section" id="filtering-tests-by-their-description">
<h2>Filtering tests by their description<a class="headerlink" href="#filtering-tests-by-their-description" title="Permalink to this headline">¶</a></h2>
<p>When a few jobs fail and need to be run again, the <code class="docutils literal notranslate"><span class="pre">--filter</span></code> option
can be used to select tests with a matching description. For instance, if the
<code class="docutils literal notranslate"><span class="pre">rados</span></code> suite fails the <a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml">all/peer.yaml</a> test, the following will only
run the tests that contain this file</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite rados --filter all/peer.yaml</span>
</pre></div></div><p>The <code class="docutils literal notranslate"><span class="pre">--filter-out</span></code> option does the opposite (it matches tests that do <cite>not</cite>
contain a given string), and can be combined with the <code class="docutils literal notranslate"><span class="pre">--filter</span></code> option.</p>
<p>Both <code class="docutils literal notranslate"><span class="pre">--filter</span></code> and <code class="docutils literal notranslate"><span class="pre">--filter-out</span></code> take a comma-separated list of strings
(which means the comma character is implicitly forbidden in filenames found in
the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">ceph/qa sub-directory</a>). For instance</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite rados --filter all/peer.yaml,all/rest-api.yaml</span>
</pre></div></div><p>will run tests that contain either
<a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml">all/peer.yaml</a>
or
<a class="reference external" href="https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/rest-api.yaml">all/rest-api.yaml</a></p>
<p>Each string is looked up anywhere in the test description and has to
be an exact match: they are not regular expressions.</p>
</div>
<div class="section" id="reducing-the-number-of-tests">
<h2>Reducing the number of tests<a class="headerlink" href="#reducing-the-number-of-tests" title="Permalink to this headline">¶</a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">rados</span></code> suite generates tens or even hundreds of thousands of tests out
of a few hundred files. This happens because teuthology constructs test
matrices from subdirectories wherever it encounters a file named <code class="docutils literal notranslate"><span class="pre">%</span></code>. For
instance, all tests in the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic">rados/basic suite</a> run with
different messenger types: <code class="docutils literal notranslate"><span class="pre">simple</span></code>, <code class="docutils literal notranslate"><span class="pre">async</span></code> and <code class="docutils literal notranslate"><span class="pre">random</span></code>, because they
are combined (via the special file <code class="docutils literal notranslate"><span class="pre">%</span></code>) with the <a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic/msgr">msgr directory</a></p>
<p>All integration tests are required to be run before a Ceph release is
published. When merely verifying whether a contribution can be merged without
risking a trivial regression, it is enough to run a subset. The <code class="docutils literal notranslate"><span class="pre">--subset</span></code>
option can be used to reduce the number of tests that are triggered. For
instance</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite --machine-type smithi --suite rados --subset <span class="m">0</span>/4000</span>
</pre></div></div><p>will run as few tests as possible. The tradeoff in this case is that
not all combinations of test variations will together,
but no matter how small a ratio is provided in the <code class="docutils literal notranslate"><span class="pre">--subset</span></code>,
teuthology will still ensure that all files in the suite are in at
least one test. Understanding the actual logic that drives this
requires reading the teuthology source code.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">--limit</span></code> option only runs the first <code class="docutils literal notranslate"><span class="pre">N</span></code> tests in the suite:
this is rarely useful, however, because there is no way to control which
test will be first.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../tests-integration-testing-teuthology-workflow/" class="btn btn-neutral float-right" title="Integration Tests using Teuthology Workflow" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../" class="btn btn-neutral float-left" title="Teuthology User Guide" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>